feat(pricing): in-process refresh + report how each price resolved - #92
Merged
Conversation
added 3 commits
July 25, 2026 05:36
The claude-opus-5 incident was not a stale cache file — it was that a running process never reloads. ensurePricingLoaded() returns early once state.loaded is true, so the fetcher's 24h TTL only ever chose which snapshot to load at startup. The dashboard LaunchAgent had been up 21 hours; it showed 3.87M tokens at $0.000000 with no signal, and nothing short of a restart would have changed that. Three defects, one fix, because lookupPricing already computes everything needed and getModelPricing was throwing it away. - Reload in the background when a lookup misses or the snapshot has aged past its TTL. Single-flight, never awaited by the request, with a 5-minute cooldown so a model that is genuinely absent upstream cannot turn every row into a fetch. A failed reload keeps the snapshot it has. - Add getModelPricingMeta(), which returns the resolution tier alongside the price. getModelPricing keeps its bare-numbers contract. - Warn once per unknown model per process, naming the model and saying its cost is being counted as $0. - Expose pricing_tier per model plus unpriced_models / fuzzy_priced_models and the snapshot's age and source on the model-breakdown response. - Dashboard: prefer the server's tier over the cost<=0 guess, which cannot tell an unpriced model from a genuinely free one, and surface fuzzy prices — those were invisible because a guessed price is never $0 and so never looked wrong. The heuristic stays as the fallback for an older server response. Also adds forceRefresh to loadLitellmData. `ttlMs: 0` looks like it forces a refetch and does not: mtime carries sub-millisecond precision that Date.now() lacks, so a cache file written moments earlier compares as "written in the future" and still counts as fresh. A test caught the reload silently re-reading the same snapshot. Closes #90
Independent QA pass (Codex, xhigh) found the reload path re-introducing the bug it was written to fix. loadLitellmData falls back on its own — upstream, then the stale disk cache, then the bundled seed. A background refresh took whatever came back and installed it. So with the disk cache gone (deleting it is the documented workaround for this very bug) and upstream unreachable, a refresh would replace good in-memory data with the OLDER seed. Reproduced: a model priced at $5/$25 dropped to $0 after one failed refresh. - Background reloads now only install data that actually came from upstream; anything else is reported and discarded. The initial load still accepts any source, because at that point there is nothing better to keep. - Report only the error CODE, never the message. fs and fetch errors carry absolute paths, and this string is served over HTTP to the dashboard. - Key the tier map by source+model like the lookup itself. Keyed by model alone, one provider's exact hit could hide another provider's miss — Antigravity normalises model names before lookup, so collisions are real. - The old failed-refresh test left the disk cache in place, so the fetcher re-read the same data and the test passed while proving nothing. It now deletes the cache, which is what makes the downgrade observable, and asserts the prices survive. Added a test that the reported error carries no path. - Mirror pricing_tier in the vite dev mock, or `dashboard:dev` cannot exercise the unpriced/fuzzy badges at all.
pitimon
pushed a commit
that referenced
this pull request
Jul 24, 2026
Independent QA pass (Codex, xhigh) on the merged preview of #91 + #92. - Refuse absolute-form request targets. Host said loopback while the target carried its own authority, and routing parses the absolute URL — so the allowlist and the router disagreed about which site the request was for. Not reachable from a browser (absolute-form only goes to proxies), but a parser differential is not something to leave open in the one guard that stands between a rebound page and the whole spend history. - Allow the fully-qualified loopback spelling. WHATWG URL canonicalises a trailing dot away for IPv4 literals but not for names, so `localhost.` got a spurious 403 while `127.0.0.1.` passed. - Refuse userinfo in a Host header. `evil.example@127.0.0.1` was accepted; the origin genuinely is loopback so this was never a bypass, but Host has no userinfo component and anything carrying one is malformed. - Scan _meta values, not key names, for unenforced dates. Matching only `*_expiry` meant `promo_cutover: "2026-08-31 — update the price"` sailed past and would have expired in silence — the exact failure the validator exists to prevent. Now any YYYY-MM-DD parked anywhere in _meta (including nested) must live in the expiries array.
added 3 commits
July 25, 2026 06:31
…ource keying
QA re-check found blocker 5 still open and blocker 3 untested.
- `e.code || e.name` was interpolated straight into a field served over HTTP.
Those are normally short symbols, but a rejected promise can carry any
object — `{code: "/Users/alice/private/pricing.json"}` reached the dashboard
verbatim. Accept only symbol-shaped values, fall back to a constant.
- The sanitizer is unit-tested directly rather than through the reload.
loadLitellmData recovers from a failed fetch on its own, so driving it end to
end only ever exercises the "fell-back" branch: an integration test asserting
"no slash" passed without touching the sanitizer at all. The end-to-end test
stays, but it is no longer the proof.
- Added the two-source collision test the re-check flagged as missing: the same
model id resolving exactly for one source and missing for another must keep
both verdicts. Reverting the source-aware keying now fails a test.
QA re-check broke the regex sanitizer in one line: `sk_live_` plus 24 characters is symbol-shaped too, so `refresh-failed:sk_live_AAAA…` was reachable. No pattern separates "a short error symbol" from "a short secret" — only an allowlist does. - Replace the regex with two closed sets: known fs/network error codes and error class names, and the four sources loadLitellmData can report. Anything else becomes "unknown". - Unit-test the allowlist against a token, a path, a GitHub-style key and an object with a lying toString; assert the real codes still pass through. - The end-to-end test now asserts the reported value is a member of the full allowed set rather than merely "contains no slash". - Removed a test for scheduleReload's catch branch. It could not be driven deterministically — loadLitellmData recovers internally and only throws when cachePath is falsy, which loadInto substitutes away — and the attempt made a real network call while asserting nothing. The gap is recorded as a comment instead of covered by a test that cannot fail.
…anch The QA gate flagged the comment as too strong: statSafe rethrows a non-ENOENT stat error, so an unusable cache path does reach scheduleReload's catch (probe observed refresh-failed:TypeError). Sanitization held, but the comment claimed the branch was practically unreachable. Record the real gap instead.
pitimon
added a commit
that referenced
this pull request
Jul 25, 2026
…n Host + sync lock (#91) * fix(pricing): apply post-promo DeepSeek rates, machine-check curated expiries deepseek-v4-pro was still pinned to its 75%-off launch promo 55 days after the promo ended, so every DeepSeek row billed at 25% of true cost. The file documented the cutover date itself, in prose, and nothing read it. A stale price is worse than a missing one: a missing price shows $0 and looks broken, a stale price looks fine forever. - Correct deepseek-v4-pro to the standard rates (1.74 / 3.48 / 0.0145 / 1.74) - Replace the free-text _meta.*_expiry keys with a structured _meta.expiries array (id / expires_at / what / action), carrying the Sonnet 5 2026-08-31 cutover that was about to repeat the same failure - Add validate:curated-expiry to ci:local so a PR opened on or after an expiry date fails until a human applies the action and clears the entry; it also rejects any regression to free-text *_expiry keys - Stop duplicating price literals in model-breakdown's coverage test — it asserts the lookup path (aliases, prefixes, casing) and reads expected rates from the curated table, so a legitimate price fix touches one file Closes #87 * fix(serve,sync): reject rebound Host headers, heartbeat the sync lock Two small hardening fixes for windows that are cheap to close now and expensive to diagnose afterwards. Host header (#88): binding to loopback does not make the Host header trustworthy. Under DNS rebinding a browser sends Host: attacker.example to 127.0.0.1 and treats the response as same-origin, so CORS never applies. Mutations were already gated on a loopback Origin, but every GET /functions/* endpoint — full spend history, model mix, project names — was readable by any page the victim had open. Requests whose Host is not loopback now get a 403 before any routing. isLoopbackHostname is reused from local-api so the Host allowlist and the Origin allowlist cannot drift. The request handler moved out of cmdServe into createRequestHandler so the test boots a real server and asserts the API handler is never reached — a guard that exists but is never wired is exactly the failure being prevented. Sync lock (#89): the lock had a 5-minute staleness window and never refreshed its own mtime, while local-sync fires on an interval. Any sync longer than one tick — full-corpus rebuilds and migration reparses are — had its lock stolen, letting two writers interleave appends into queue.jsonl. A torn line is silently skipped by the reader, and a skipped retraction row is a permanent overcount. - Heartbeat the lock mtime every 30s (unref'd, cleared on release) and raise the stale window to 30 minutes: "stale" now means the holder died - Record pid/host/startedAt in the lock, and reclaim immediately when the recorded process is gone — faster recovery than the old window, not slower - Gate the stale takeover behind an atomic mkdir mutex. The previous check-then-act let two waiters both delete and both acquire; a rename-based claim was tried first and still failed a 4-way race test, because rename is atomic but does not bind the check to the act - Clean up a failed lock write instead of leaking the fd and an empty file Closes #88 Closes #89 * fix(serve,pricing): close QA findings on the Host guard and expiry check Independent QA pass (Codex, xhigh) on the merged preview of #91 + #92. - Refuse absolute-form request targets. Host said loopback while the target carried its own authority, and routing parses the absolute URL — so the allowlist and the router disagreed about which site the request was for. Not reachable from a browser (absolute-form only goes to proxies), but a parser differential is not something to leave open in the one guard that stands between a rebound page and the whole spend history. - Allow the fully-qualified loopback spelling. WHATWG URL canonicalises a trailing dot away for IPv4 literals but not for names, so `localhost.` got a spurious 403 while `127.0.0.1.` passed. - Refuse userinfo in a Host header. `evil.example@127.0.0.1` was accepted; the origin genuinely is loopback so this was never a bypass, but Host has no userinfo component and anything carrying one is malformed. - Scan _meta values, not key names, for unenforced dates. Matching only `*_expiry` meant `promo_cutover: "2026-08-31 — update the price"` sailed past and would have expired in silence — the exact failure the validator exists to prevent. Now any YYYY-MM-DD parked anywhere in _meta (including nested) must live in the expiries array. * fix(serve): treat empty userinfo as userinfo, refuse network-path targets QA re-check found two gaps in the previous round's guard. - "@localhost" and ":@localhost" parse to a falsy url.username, so checking the parsed fields let exactly the malformed forms through while the fully-spelled "user:pass@localhost" was refused. Test the raw header for "@" instead. - "//evil/x" and "/\\evil/x" start with a slash and so passed the origin-form check, but WHATWG URL resolves both against a foreign authority (new URL("/\\evil/x", "http://localhost").hostname === "evil"). Routing only reads url.pathname today, so nothing is exploitable now — but handing a handler a URL that points at someone else's origin is the same guard-vs-parser disagreement absolute-form creates. * fix(serve): refuse control characters in the Host header and request target QA re-check found the last hole in the prefix checks: WHATWG URL strips tab, LF and CR from its input BEFORE parsing, so "/<tab>//evil/x" becomes "//evil/x" and adopts a foreign authority after passing a startsWith("//") test. Same trick applies to the Host string. Verified over a socket that Node's own parser returns 400 for those bytes in a request-target before the handler ever runs, so this was not reachable through the real server — recorded in the test so the next reader does not have to re-derive it. Fixed anyway: a guard that holds only because a different layer happens to be strict is exactly the guard-vs-parser disagreement this function exists to prevent. --------- Co-authored-by: itarun.p <itarun.p@somapait.com>
pitimon
pushed a commit
that referenced
this pull request
Jul 25, 2026
Ships the QA-gated fixes from #91 and #92: corrected DeepSeek rates with a machine-checked expiry, Host-header and sync-lock hardening, and pricing that refreshes in-process and reports how each price resolved. prepublishOnly re-vendored the LiteLLM seed, which now carries claude-opus-5 — so a cold start prices it correctly even before the first background refresh.
This was referenced Jul 25, 2026
Closed
pitimon
added a commit
that referenced
this pull request
Jul 25, 2026
Closes issue 106. README:32 already made this claim and the product did not compute it: it showed what usage WOULD cost at list price, and it showed a quota bar, but the comparison — the actual decision — was left as mental arithmetic against a plan price nothing ever asked for. Every input already existed: per-source cost per window and the pricing tier telling you how much to trust it, both already on the model-breakdown endpoint. The arithmetic is a subtraction. THE LABELLING IS THE HARD PART, which is why the logic lives in a pure module with its own tests rather than inside JSX. Three rules, all from the issue, each with a test that fails if a future edit drops it: 1. It never says "you saved". This is LIST-PRICE-EQUIVALENT, not a counterfactual bill — anyone actually on the API would use it differently. A test asserts the rendered card contains no "saved", "savings", "wasted", "worth it" or "bargain", and the disclaimer line always renders. 2. It inherits the pricing caveats from #92. If any model in the window is unpriced or fuzzy-matched, the figure is a FLOOR and says so, naming the models. `unattributed` from #94 counts as unpriced. One floor anywhere makes the whole roll-up a floor — averaging confidence would let a well-priced provider vouch for a badly-priced one. 3. Under-usage reads exactly like over-usage. "$3 of usage on a $20 plan" is a downgrade signal and is as useful as the other direction, so neither gets a colour or a verdict. Tested by asserting the below case renders the same sentence with no "over"/"under" language. The plan price is entered by the user and stored in localStorage. Never sent anywhere — no plan catalogue to maintain and go stale, and a plan price is exactly the kind of thing this product promises not to transmit. A test asserts the storage hook contains no fetch and imports no API client. Absent is never zero, in three places: no stored price means no comparison rather than a $0 plan (which is infinitely over by construction), clearing the field stores nothing rather than 0, and a provider the user has not priced is left out of the roll-up entirely rather than inflating the list-price side. I nearly shipped this without an input. The card rendered "enter what you pay" with nowhere to enter it, which is not a feature. The inputs render in BOTH states because in the empty state they are the only way out of it. README:32 rewritten to match what shipped rather than left overselling it — the issue's own instruction if the honest framing could not be made to fit. It can, so the claim now names the window, the list-price-equivalence, and the floor. 37 new tests: 14 on the pure comparison, 8 on storage (corrupt JSON, non-object, non-positive values, a write that throws), 14 on the card, plus a wiring guard. That last one is a source-fact test rather than a render test because there is no DashboardView harness — and this session already produced the lesson twice: a Vite build stayed green over a deleted identifier, and a notice went into a component nothing mounts. ci:local exit 0: 952 root tests, 302 dashboard. Co-authored-by: itarun.p <itarun.p@somapait.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes the "the dollar number is confidently wrong and nothing says so" class that produced the
claude-opus-5$0incident.The root cause was not the stale cache file.
ensurePricingLoaded()returns early oncestate.loadedis true, so a running process never reloads — the fetcher's 24h TTL only ever chose which snapshot to load at startup. The dashboard LaunchAgent had been up 21 hours. Deleting the cache was incidental; a restart alone would have fixed it, and nothing short of a restart ever would have.getModelPricingMeta()returns the resolution tier next to the price.getModelPricingkeeps its bare-numbers contract, so no existing caller changes.$0.pricing_tierper model on the model-breakdown response, plusunpriced_models,fuzzy_priced_models, and the snapshot's age/source.cost <= 0guess, and surfaces fuzzy-priced models.Two things worth knowing before reviewing
1. The "no UI badge" premise was wrong — half of this already existed.
buildFleetDataalready computespricingMissingand bothHeroSummaryandProviderBreakdownCardalready render a count chip. So during the incident the dashboard did show a missing-pricing chip; it was just weak enough that nobody looked. What was genuinely missing is that the client had to infer it fromcost <= 0, which cannot distinguish an unpriced model from a genuinely free one — hence theisKnownZeroCostModelname-sniffing hack (includes("free"),hy3-preview,glm-*-flash). The server now says which it is, and the heuristic stays only as the fallback path for an older server response.2. Fuzzy pricing had no surface at all, and it is the more dangerous half.
matcher.jsresolvesacme-9-turboagainst the literal substring keyacme-9and returns that price. The result is non-zero, so nocost <= 0heuristic can ever flag it — a guessed price never looks wrong.curated-overrides.jsonalready carries{"match": "kimi", "ref": "kimi-k2.5"}, and the file already needed akimi-k2.6exact entry to escape that net.3.
ttlMs: 0does not force a refetch. The first implementation used it, and a test caught the reload silently re-reading the same snapshot:isFreshcomparesDate.now() - stat.mtimeMs < ttlMs, and mtime carries sub-millisecond precision thatDate.now()does not — a cache file written moments earlier compares as "written in the future" and counts as fresh. Fixed with an explicitforceRefreshflag onloadLitellmData.Test plan
npm run ci:localgreen end to end (dashboard build + 248 dashboard tests + 785 root tests + all validators)$0, tiermiss), publish the model upstream, and assert it prices correctly without a restart — the negative cache must be cleared by the reloadgetModelPricingstill returns the same object asgetModelPricingMeta().pricingensurePricingLoadeddo not throw or attempt a reloadpricing_tier: "litellm:exact"with$0is not flagged as unpriced (genuinely free),pricing_tier: "miss"is, andpricing_tier: "litellm:fuzzy"with a non-zero cost is surfaced separatelycost <= 0heuristic whenpricing_tieris absent:17680thatunpriced_modelsis empty and no model row carries a fuzzy tier unexpectedlyFollow-ups not in this PR
_meta.expiriesenforcement and the loopback/lock hardening are fix: correct expired DeepSeek prices, enforce curated expiries, harden Host + sync lock #91.Closes #90